Evaluating Ranking Diversity and Summarization in Microblogs using Hashtags

Diversification techniques for web search have recently been developed that assume that, for each query, there is a set of underlying aspects or subtopics that address specific user intents. These techniques attempt to balance the relevance of the retrieved documents with the coverage of the aspects. Evaluation of diversification techniques requires some way of defining a set of aspects for each test query and a “gold standard” assignment of documents to aspects. This has made the study of diversification difficult for new data such as microblogs. A related task, keyword-based summarization, is important for microblogs but also has problems in evaluation. In this paper, we describe an approach to evaluating ranking diversity and summarization in microblogs by assuming hashtags correspond to subtopics. We show the viability of this approach to evaluation, and validate the assumption that hashtags are subtopics. The results show that, despite the differences in content, the best techniques for search diversification with microblogs are the same as with web pages. The summarization results confirm that the DSPapprox technique is effective and that phrase-based summarization techniques perform somewhat worse than single words in terms of covering the underlying aspects.